Papers with continual pre-training
HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization (2025.acl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models exhibit suboptimal performance on mid- and low-resource languages due to limited exposure to these languages during pre-training. |
| Approach: | They propose a similarity-based subword embedding initialization heuristic that introduces new tokens specific to target languages, initializes their embedders, and applies continual pre-training on target-language data. |
| Outcome: | The proposed method outperforms random initialization baseline and matches or exceeds OFA in both continual pre-training convergence and downstream task performance. |
Can Continual Pretraining Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain? (2026.acl-long)
Copied to clipboard
Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh, Hammam Abdelwahab, Héctor Allende-Cid, Katrin Klug
| Challenge: | specialized models have a large potential for translation and translation, but they lack the integration of domainspecific knowledge and terminology into clinical workflows. |
| Approach: | They construct a German medical corpus to continuously pre-train and merge three well-known LLMs and use it to improve model performance. |
| Outcome: | The proposed model family significantly outperforms the mistral-Small-24B-Instruct model family on German medical benchmarks. |
Improving Knowledge Production Efficiency With Question Answering on Conversation (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing researches on conversation-based QA focus on document-based tasks . current researche focuses on document based tasks, but there is a lack of researche on conversation based qa . |
| Approach: | They propose a multi-span extraction model on conversation-based QA and introduce continual pre-training and multi-task learning schemes to further improve model performance. |
| Outcome: | The proposed model outperforms baseline on two Chinese datasets and will be released for research purposes. |
Efficient Continual Pre-training of LLMs for Low-resource Languages (2025.naacl-industry)
Copied to clipboard
| Challenge: | Open-source large language models (LLMs) are a promising tool for low-resource languages . however, there is still a substantial performance gap between high-resourced languages and LRLs . |
| Approach: | They develop an algorithm to select a subset of texts from a larger corpus and use it to select tokens for LLMs. |
| Outcome: | The proposed algorithm reduces the cost of continual pre-training (CPT) with large amounts of language-specific data. |
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to embed multimodal models face limitations such as suboptimal causal attention in VLMs and limited diversity in training objectives and data. |
| Approach: | They propose a framework for transforming pre-trained VLMs into bidirectional multimodal embedding models. |
| Outcome: | The proposed model improves performance across MMEB and ViDoRe-v2 benchmarks and exhibits strong scalability with both model size and training data on MMEF. |
Sailor: Open Language Models for South-East Asia (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) rely on English data for training, but are often not comparable across other languages. |
| Approach: | They propose to develop a family of open language models for SEA languages . they use BPE dropout, aggressive data cleaning and deduplication to improve model robustness . |
| Outcome: | The proposed models perform well across four benchmarks, including commonsense reasoning, question answering, reading comprehension and examination. |
ReGLA: Refining Gated Linear Attention (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) are known for their computational and storage requirements due to the quadratic computation complexity of softmax attention. |
| Approach: | They propose to reduce the quadratic computation complexity of softmax attention by using feature maps, normalization and the gating mechanism to improve performance. |
| Outcome: | The proposed model outperforms existing gated linear attention models in extensive tasks including training from scratch and post-linearization with continual pre-training. |
Demystifying Mixed Outcomes of Self-Training: Pre-training Analyses on Non-Toy LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies on self-training report seemingly contradictory outcomes. |
| Approach: | They use OLMo-2 models as non-toy LLMs and perform multiple rounds of continual pre-training using self-generated text with different prompting strategies and data filtering. |
| Outcome: | The proposed model collapse is inherent to the training procedure itself, while self-improvement is likely owes its success to human-designed, strategic synthetic pipelines that inject external intelligence. |
Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | a lack of data in low-resource languages has limited the performance of a multilingual pre-trained model. |
| Approach: | They propose a continuous pre-training framework to adapt mBART to unseen languages . they construct noisy mixed-language text from the monolingual corpus of the target language . |
| Outcome: | The proposed framework improves finetuning performance on low-resource translation pairs . the proposed framework also improves on translation pairs where both languages are seen . |
CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data (2024.lrec-main)
Copied to clipboard
| Challenge: | Clinical data in hospitals are unstructured and therefore need to be extracted from medical reports to conduct clinical studies. |
| Approach: | They propose a dedicated French biomedical model based on a public French biomedicine dataset. |
| Outcome: | The proposed model improves 2.54 points of F1-score on biomedical named entity recognition tasks. |
Vicinal Risk Minimization for Few-Shot Cross-lingual Transfer in Abusive Language Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for few-shot cross-lingual transfer learning are limited in target languages due to the scarcity of resources. |
| Approach: | They propose a method which interpolates pairs of instances based on the angle of their representations and propose augmentation methods to enhance few-shot cross-lingual abusive language detection. |
| Outcome: | The proposed method improves few-shot cross-lingual abusive language detection in seven languages typologically distinct from English and three different domains. |
ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)
Copied to clipboard
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Improving Referring Ability for Biomedical Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing auto-regressive large language models (LLMs) are primarily trained using documents from general domains. |
| Approach: | They propose to use citation network to improve the pre-training of auto-regressive large language models (LLMs) in the biomedical domain. |
| Outcome: | Empirical studies show that the proposed method improves both the intra-sample and inter-sammple referring abilities of auto-regressive large language models in the biomedical domain. |
Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for reweighting data mixtures rely on manual designation with certain heuristics based on intuition or empirical results. |
| Approach: | They propose a model-based framework that learns to re-weight domains by reinforcement learning on large quantities of data mixing trajectories with corresponding feedback from an evaluation environment. |
| Outcome: | The proposed framework outperforms baselines in achieving balanced performance across source and target fields and domain spaces without retraining. |
Improving Relation Extraction by Sequence-to-sequence-based Dependency Parsing Pre-training (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies show that dependency information is used only for encoder-only-based relation extraction tasks. |
| Approach: | They propose a syntax-aware seq2seq pre-trained model for relation extraction that incorporates dependency information into a seq2-trained language model by continual pre-training with a dependency parsing task. |
| Outcome: | The proposed model incorporates dependency information into a seq2seq pre-trained language model by continual pre-training with a generative sequence-to-sequence (sequ2sq)-based dependency parsing task. |
Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical studies show that learning multiple training objectives in a single model makes the learned language representation barely converge to the desired optimum. |
| Approach: | They propose a meta-learning-based adaptive sampler which learns latent sampling pattern on arbitrary pre-training objectives. |
| Outcome: | Empirical studies show that learning multiple objectives in a single model makes it difficult to achieve the desired optimum. |
Reinforcement Learning on Pre-Training Data (2026.acl-long)
Copied to clipboard
Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Kun Li, Haoyuan Wu, null Wujiajia, Zihao Zheng, Chenchen Zhang, Kun Shi, Xue Gong, Qi Yi, Ruibin Xiong, Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang xu, Zheng Fang, Shuai LI, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bochao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Bo Zhou, Di Wang
| Challenge: | Recent progress in large language models is driven by scaling of training compute through pre-training with nexttoken prediction (NTP) or post-training (RL) Pre-training using NTP enables models to acquire extensive knowledge and skills from general data, but it suffers from data inefficiency and catastrophic forgetting in continual learning settings. |
| Approach: | They propose to scale training compute through pre-training with next-token prediction (NTP) or post-training by scaling reinforcement learning (RL) to improve learning from general data. |
| Outcome: | Experiments on multiple benchmarks and models show that the proposed approach improves continual pre-training and provides a strong foundation for post-training on Qwen3-8B-Base. |
LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have extended context windows through scaling positional encodings and lightweight continual pre-training, but performance degradation is still not fully explored. |
| Approach: | They propose a novel approach to reduce short-text performance degradation by minimizing distribution drift in hidden states and attention scores. |
| Outcome: | The proposed approach minimizes the distribution discrepancy between the extended and original models while maintaining or even enhancing the model's long-context abilities. |
Efficient Continual Pre-training for Building Domain Specific Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are typically trained entirely on domain corpus to excel at handling domain-specific tasks. |
| Approach: | They propose a continual pre-training strategy to build domain-specific LLMs over existing open-domain LLM. |
| Outcome: | The proposed model outperforms existing LLMs with 10% of corpus size and cost without any degradation on open-domain tasks. |
Efficient Training for Cross-lingual Speech Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) focus on the text modality, making speech modeling difficult. |
| Approach: | They propose a cross-lingual speech language model that trains on discrete speech tokens to achieve cross-modal and cross-linguistic alignment through continual pre-training. |
| Outcome: | The proposed method achieves cross-modal and cross-lingual alignment through continual pre-training. |
Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities when leveraging in-context learning. |
| Approach: | They propose a method that discretizes uninformative tokens using a self-supervised pre-training technique. |
| Outcome: | The proposed method achieves state-of-the-art performance across classification tasks while requiring only 0.8% decrease in performance. |
A Two-Stage Adaptation of Large Language Models for Text Ranking (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models (PLMs) have significantly improved ranking performance in text ranking tasks. |
| Approach: | They propose a two-stage progressive paradigm to better adapt LLMs to text ranking by conducting continual pre-training on a large weakly-supervised corpus and performing SFT on high-quality data. |
| Outcome: | The proposed approach outperforms previous methods on in- and out-domain scenarios. |
Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent progress in intent detection relies on deep models and datasets with well-crafted annotations. |
| Approach: | They propose a continual pre-training approach to train deep learning models . they propose augmentation method and sequential self-distillation to boost performance . |
| Outcome: | The proposed method outperforms methods that employ continual pre-training on labeled datasets on few-shot intent detection tasks. |
Recyclable Tuning for Continual Pre-training (2023.findings-acl)
Copied to clipboard
Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou
| Challenge: | Continual pre-training is the paradigm where pre-trained language models acquire fresh knowledge and gradually get upgraded. |
| Approach: | They propose to use adapted weights to recycle old PLMs for continual pre-training . they propose to combine initialization and distillation methods to achieve better performance . |
| Outcome: | The proposed method improves the convergence and performance of the upgraded PLM. |
A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models (2024.emnlp-main)
Copied to clipboard
Zhihao Wang, Shiyu Liu, Jianheng Huang, Wang Zheng, YiXuan Liao, Xiaoxin Chen, Junfeng Yao, Jinsong Su
| Challenge: | Version updates are an indispensable requirement for Large Language Models . a large learning rate in the first stage and a complete learning decay process are crucial for version updates of LLMs. |
| Approach: | They propose a learning rate path switching training paradigm for version updates of Large Language Models. |
| Outcome: | The proposed paradigm reduces training cost to 58% when training four versions of LLMs compared to PTFS and CPT . |
Improving Continual Pre-training Through Seamless Data Packing (2025.findings-acl)
Copied to clipboard
| Challenge: | Empirical evaluations across various model architectures and corpus domains demonstrate the effectiveness of our method, outperforming baselines in 99% of all settings. |
| Approach: | They propose a method that uses a sliding window technique to pack data before continual pre-training to preserve contextual information and enhance model performance. |
| Outcome: | Empirical evaluations across various model architectures and corpus domains demonstrate the effectiveness of the proposed method outperforming baselines in 99% of settings. |
AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text (2025.findings-emnlp)
Copied to clipboard
Tadesse Destaw Belay, Israel Abebe Azime, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Idris Abdulmumin, Abinew Ali Ayele, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam
| Challenge: | Domain adaptive pre-training and task-adaptive pre- training (TAPT) are popular methods to reduce this bias for low-resource languages, but they have not been explored for African multilingual encoders. |
| Approach: | They propose a large-scale social media and news domain corpus for continual pre-training on African languages. |
| Outcome: | The proposed methods improve performance on three subjective tasks, including sentiment analysis, multi-label emotion, and hate speech classification, while TAPT improves performance on other related tasks. |
LexMatcher: Dictionary-centric Data Curation for LLM-based Machine Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) has brought about new opportunities for machine translation. |
| Approach: | They propose a method for data curation that supplements the infrequent senses of polysemous words. |
| Outcome: | The proposed method outperforms established baselines on the WMT2022 test sets and is applicable to other pre-trained models. |
Sparsity-Accelerated Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated proficiency across various NLP tasks but often require additional training, such as continual pre-training and supervised fine-tuning. |
| Approach: | They propose to leverage sparsity in pre-trained LLMs to accelerate training by disregarding computations for unimportant neurons. |
| Outcome: | The proposed framework achieves comparable or superior performance to standard training while significantly accelerating the process. |
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) has gained increasing popularity as a framework for scaling up large language models. |
| Approach: | They investigate how to build Mixture-of-Experts (MoE) models from existing large language models . they use expert construction, Continual pre-training and data sampling strategies . |
| Outcome: | The proposed model outperforms existing models with similar parameters on a wide range of tasks. |
Data-Centric Continual Pre-training for 500+ Languages: A New Bilingual Translation Corpus and Multilingual Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models pre-trained on massive data have promoted multilingual natural language processing (NLP). |
| Approach: | They construct a bilingual translation corpus with 2,500 language pairs and develop a suite of four models with parallel data. |
| Outcome: | The proposed model suites are evaluated across 7 tasks and 12 benchmarks. |
LESA: Learnable LLM Layer Scaling-Up (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for depth scaling-up rely on empirical heuristic rules for layer duplication, resulting in poor initialization and slower convergence during continual pre-training. |
| Approach: | They propose a method for learning latent parameters between layers by concatenating parameters from each layer and applying Singular Value Decomposition. |
| Outcome: | Experiments show that LESA outperforms baseline models with less than half the cost of existing methods. |
Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models (2025.findings-emnlp)
Copied to clipboard
Xiaojun Wu, Junxi Liu, Huan-Yi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, Jian Guo
| Challenge: | Existing financial benchmarks suffer from limited language and task coverage, low-quality datasets, and inadequate adaptability for LLM evaluation. |
| Approach: | They propose a bilingual benchmark for financial LLMs that assesses models’ language understanding and generation capabilities. |
| Outcome: | The proposed bilingual benchmark assesses models’ language understanding and generation capabilities. |
Assessing the Belief Consistency of Large Language Models on the Logical Conversation Process (2026.acl-long)
Copied to clipboard
Tomoki Tsujimura, Matīss Rikters, Masaki Asada, Shusaku Egami, Tatsuya Ishigaki, Ken Yano, Hiroya Takamura
| Challenge: | Large language models have been shown remarkable ability to understand given contexts. |
| Approach: | They propose a method to evaluate whether beliefs held by LLMs remain consistent . they propose to use multiple choice question answering format to assess belief consistency . |
| Outcome: | The proposed method evaluates the consistency of LLMs in a multiple-choice question answering format. |
From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan (2026.acl-long)
Copied to clipboard
Lei Yang, Leiyu Pan, Bojian Xiong, Renren Jin, Shaowei Zhang, Yue Chen, Ling Shi, Jiang Zhou, Junru Wu, Zhen Wang, Jianxiang Peng, Juesi Xiao, Tianyu Dong, Zhuowen Han, Zhuo Chen, Yuqi Ren, Deyi Xiong
| Challenge: | Large language models have achieved remarkable success across a wide range of tasks, yet their performance remains heavily biased toward high-resource languages. |
| Approach: | They propose a pipeline for advancing Tibetan language modeling through multilingual continual pre-training with Tibetan, Chinese, and English. |
| Outcome: | The proposed model outperforms open-source and Tibetan-focused models on diverse tasks. |